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ABSTRACT 

Order picking accounts for 55% of the annual $60 billion 
spent on warehouse operations in the United States. Reduc- 
ing human-induced errors in the order fulfillment process can 
save warehouses and distributors significant costs. We investi- 
gate a radio-frequency identification (RFID)-based verification 
method wherein wearable RFID scanners, worn on the wrists, 
scan passive RFID tags mounted on an item’s bin as the item 
is picked; this method is used in conjunction with a head-up 
display (HUD) to guide the user to the correct item. We com- 
pare this RFID verification method to pick-to-light with button 
verification, pick-to-paper with barcode verification, and pick- 
to-paper with no verification. We find that pick-to-HUD with 
RFID verification enables significantly faster picking, provides 
the lowest error rate, and provides the lowest task workload. 


ACM Classification Keywords 
H.5.2. Information interfaces and presentation (e.g., HCD: 
Miscellaneous; User Interfaces-Evaluation/methodology 


Author Keywords 
Order Picking; Wearable Computers; Head-Up Display; 
RFID; Augmented Reality 


INTRODUCTION AND RELATED WORK 
Order picking, the process of collecting items in a specified 
quantity to fulfill a customer's order, accounts for over $30 
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billion in annual warehousing expenditures in the United 
States alone [2, 3]. Currently, 80% of order picking is per- 
formed by humans using paper-based pick lists [8]. Previous 
research has studied the impact of various technologies in 
order picking. Weaver et al. compared pick-to- head-up 
display (pick-to-HUD) to pick-to-voice and found HUDs to 
be significantly faster [11]. Guo et al. compared pick-to-paper, 
pick-to-light, and pick-to-HUD [5]. Pick-to-HUD was faster 
than the other methods. Other studies have reached similar 
conclusions [1, 13]. 


While HUD has already been shown to have major improve- 
ments over other methods in terms of efficiency, accuracy, and 
comfort, previous studies found that HUD systems trended 
toward more errors than pick-to-light systems [13]; however, 
the results were not statistically significant. Wu et al. man- 
aged to reduce errors in pick-to-HUD by using a weight-based 
verification system [12]. Industrial weight systems, however, 
typically require the picker to place one item on the scale at a 
time, which impacts the HUD’s speed advantage. Iben et al. 
implemented pick verification by coupling laser rangefinders 
with a HUD [6]. The combined system, however, was not 
an improvement over pick-to-light with button verification as 
the picker tended to brush through the rangefinder's region 
creating false triggers. 


We improve upon pick-to-HUD order picking by using two 
wearable RFID readers, attached to the picker’s wrists, which 
detect passive tags placed in individual bins. These wearable 
bands allow real-time pick verification while keeping both 
hands free for the task. 


Radio Frequency Identification (RFID) technology 
Although RFID technology has been mature for over a decade, 
the largest barrier to adoption has been the unit cost of passive 


RFID tags which can be up to $0.10 [10]. For an RFID- 
based verification method to be useful, warehouses must install 
passive RFID tags on every item or every pick bin. Printed 
barcode labels, in contrast, cost less but require large enough 
surface area so that the barcodes can be read at a distance. For 
an automobile pick line with 1000 bins of items, instrumenting 
every bin with a passive RFID tag would cost about $700 more 
than the cost of the equivalent barcodes, suggesting that the 
functional cost savings may well be worth this initial overhead. 


Pick and Verification Methods 

We test four order picking methods: pick-to-paper with no veri- 
fication (the industry default of using printed sheets specifying 
item numbers and quantities), pick-to-paper where the picker 
verifies a pick by scanning a barcode on the item’s bin, pick- 
to-light where the picker confirms a pick by pressing a button 
mounted in front of the item’s bin, and pick-to-HUD where ver- 
ification occurs when the picker’s wrist passes a passive RFID 
tag mounted on the item’s bin. We have chosen to use a short- 
hand notation to reference these pick methods: <guidance 
mechanism>-<error verification mechanism>; therefore, we 
refer to the aforementioned methods respectively as Paper- 
None, Paper-Barcode, Light-Button, and HUD-RFID. 


Pick-to-Paper with No Verification (Paper-None) 
Pick-to-paper with no verification is the most rudimentary 
approach to order picking and serves as the baseline for com- 
parison against other order picking methods. The approach is 
very popular [9] as it is intuitive to most pickers and requires 
very little training or upfront monetary investment other than 
the cost of the paper itself. The lack of a verification method, 
however, introduces significant concerns. Picking accuracy is 
solely based on the picker's attentiveness, which is burdened 
by the constant need to shift the attentional and physical focus 
of the eyes from paper to the environment. 


Pick-to-Paper with Barcode Verification (Paper-Barcode) 

Adding a simple barcode scanner to pick-to-paper has become 
an industry staple for verifying picks. There are many forms 
of barcodes used in warehouses today including 1D (e.g. UPC, 
EAN, GS1, etc.), 2D (e.g. QR Code, Datamatrix, etc.), and 
more recently 3D barcodes. Barcode scanners incur a cost 
to the speed of order picking, but they also provide potential 
advantages to the warehouse. First, they reduce the number of 
incorrect picks, as the picker is required to verify each item 
with a scan. Second, they allow the warehouse management 
system (WMS) to be aware of the last-scanned location of the 
picker, which enables the implementation of more advanced 
optimization methods, like task interleaving, path finding, etc. 


Pick-to-Light with Button Verification (Light-Button) 

In pick-to-light, pickers receive the picking information via 
small LED displays attached to each bin. Pick-to-light without 
verification has been shown to be significantly faster than pick- 
to-paper [5]. The addition of button verification may have a 
negative effect on speed, but it greatly reduces the number 
of errors [13], making it one of least error-prone methods 
available [12]. 


Pick-to-light is commonly implemented in dense picking envi- 
ronments, where the high installation cost ($100-130 per pick 


Figure 1. Android phone, Google Glass, and wearable RFID readers 


location) is counterbalanced by the large number of picks per 
unit of distance. Because of the hardware and the wiring, pick- 
to-light is not ideal for warehouses that require frequent layout 
rearrangements, and it can be difficult to support simultaneous 
picking by several pickers. 


Pick-to-HUD with RFID Verification (HUD-RFID) 
Pick-to-HUD has been repeatedly [5, 11, 12, 13] shown to be 
the fastest method in order picking. Previous authors [5, 12, 
13], have attempted to measure HUD’s accuracy compared 
to pick-to-light, and while they have found HUDs to trend 
higher in errors, the results are not statistically significant due 
to the small number of overall errors. For this experiment, we 
designed our pick tasks to elicit as many errors as possible; 
our primary goal is to study errors while maintaining speed. 
While this setup is different than previous studies, we can 
still compare HUD and RFID independently by comparing 
to the Paper-None baseline. In this work, we propose a so- 
lution which combines the speed of pick-to-HUD with the 
verification ability of wearable RFID readers. 


IMPLEMENTATION 

We used Google Glass for our Heads-Up Display and Ubimax 
xBands for our wearable RFID readers. We also used a mobile 
device, the Samsung Galaxy A3 (2016), to manage and oversee 
the experiment. (Figure 1) 


The two wearable RFID readers were connected via a Blue- 
tooth Low Energy (BLE) session to the mobile device which 
was in turn connected to the HUD. When a user reached into 
a bin, the band would read the passive RFID tag located on 
the lip of the bin and would send the encoded bin tag string to 
the mobile device. Adjusting the reader's power was a crucial 
factor for the effectiveness of our method. Too high a power, 
and the readers would pick up the nearby tags as well. Too low, 
and the subjects would struggle to get a valid scan, making the 
whole experience much less smooth. 


The mobile device would receive the scan, decide whether it 
was correct, and would update the HUD user interface appro- 
priately. The user would also hear either a confirmation or 
error tone after each pick. 


User Interface 
In order for HUD applications to be successful, special care 
has to be given to the way the information is displayed. We 
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used a rapid iteration approach, designing our UI based on 
user feedback. Are users getting the messages we want them 
to? To achieve that, we attempted to create a UI that closely 
resembled the physical world (Figure 2). 


We color-coded our UI elements with the same color-pattern 
we use on our racks. We used a white tag on the top of the 
screen to identify the active rack and added faded-out gray 
cells on the side to help the user intuitively find the active rack. 


Error Recovery with HUD 

Whenever the wrong pick was detected, the user was sent an 
alert. There were three types of error alerts: incorrect item 
picked, incorrect rack, and incorrect receive bin. All error 
alerts where accompanied by a negative sound stimulus. 


Incorrect Item Picked 

In this case, the UI draws a small red X on the incorrect bin 
scanned. If the user scans the incorrect bin again, the small 
red X is removed. 


Incorrect Rack 
If the user scans an incorrect rack (e.g. B instead of A), the UI 
flashes a red X briefly as displayed in Figure 3. 


Incorrect Receive Bin 

This error occurs when the picker places the items in the wrong 
receive bin. Here, the UI would show the items that were 
incorrectly placed and their respective quantities. It would 
also indicate the correct receive bin as shown in Figure 4. The 
user would tap the Glass to dismiss. 


ENVIRONMENT 

We utilize a similar environment to Wu et al. [12] featuring a 
dense picking environment with two 4 x 3 matrices (“racks’’) 
of source bins (each containing about 50 instances of house- 
hold items like batteries or paperclips). These two racks were 
positioned to the left and in front of a cart containing three 
receive bins. Each source bin is outfitted with a label indicat- 
ing its position in the rack matrix, a printed barcode encoding 
this unique position, a seven-segment LED and button confir- 
mation device (both used solely in pick-to-light - see Figure 
5), and a passive RFID tag encoding the position information. 
See Figures 6 and 7. 


In the pick-to-light trials, these two racks are connected 
through Ethernet interfaces to a laptop controlling the display 
of the LEDs and responding to the push of the confirmation 
buttons. Figure 8 illustrates how the different physical, hard- 
ware, and software components of our setup interact. 


In the pick-to-barcode trials where barcode verification is 
used, the wireless barcode scanner is connected directly to a 
computer handling scans of the source and receive bins. Upon 
scan of the wrong source or receive bin, the computer plays an 
error tone through an external stereo speaker clearly audible 
to the user, informing them of their error. 


In the pick-to-HUD trials where RFID verification is used, 
the LED light array and barcode error tone systems are not 
utilized. 


We define the following terms to describe our setup: 


Task 5 - Order 1 


Figure 2. The left hand side of the screen corresponds to the active rack, 
while the right hand side of the screen corresponds to the cart. 


Task 5 - Order 1 


Figure 3. A screenshot of the UI, showcasing an incorrect rack error. 
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Task 5 - Order 3 


Figure 4. When the user places the picked items in the wrong receive bin, 
the heads up display will inform the user how to correct their mistake. 


oo | 
TO | 32 | 
ReLLAas2 


Figure 5. A source bin with identification methods. From top to bottom: 
a passive RFID, a two-digit seven-segment LED, a visual “ 32” and a 
printed barcode. 


Item - A single item. 


Source Bin - The bin from which the picker picks a number 
of items. 


e Receive Bin - The bin where the picker puts the items. 


Rack - A group of source bins. (A or B - 4 x 3). 


e Cart - A group of receive bins (1 x 3) mounted on a wheeled 
cart. 


We also define the following subdivisions for a task: 
e Pick - An acquisition of items from a source bin. 


e Place - An unloading of items into a receive bin. 


Suborder - Picking all the items from a single rack, and 
placing them in a single receive bin. 


Order - An order is housed in a single receive bin and is 
what will go to the end customer. Each order is a set of two 
suborders. 


Task - A task is a set of three orders; one for each receive 
bin. 


A single task was conducted as follows: 


1. Pick items from rack A and place them in the first receive 
bin, to complete the first suborder. 


2. Proceed to fulfill the next two suborders and then move on 
to the other rack and repeat the process. 


3. Once all 6 suborders have been conducted, A1-3 and B1-3, 
the task is completed. 


STUDY METHODOLOGY 
Based on previous research, we held the following a priori 
hypotheses: 


e H1: Average task time of HUD-RFID is less than average 
task time of all other methods 


H2: Average item error of HUD-RFID is less than average 
item error of all other methods 


H3: HUD-RFID has lower subjective workload than all 
other methods 


e H4: HUD-RFID is overall preferred over all other methods 


We conducted a within-subjects user study to evaluate the 
pick methods. Our study consisted of 12 participants (eight 
male, four female). All participants were right-hand dominant. 
Nine participants were right-eye dominant and three were left- 
eye dominant. All participants were first-time order pickers. 
We counterbalanced our conditions using a 4-by-4 balanced 
Latin square to help avoid ordering effects. Participants were 
compensated $30 for their participation in the two hour study 
and were instructed to perform the picks as quickly and as 
accurately as possible. 


Each subject would first go through a training session of five 
tasks per method in order to help extinguish learning effects 


(Figure 9). During training, researchers would actively assist 
participants in answering any questions they had about the 
method they were running. 


Ten tasks per method were utilized in the testing phase. In 
order to induce more errors than prior work [12], the number of 
picks in each suborder was selected in a two phase mechanism. 
First the number of source bins per order was selected from a 
non-uniform distribution where the probability selecting four 
bins was 90%, five bins was 5%, and six bins was 5%. Second, 
the number of items per each selected source bin was selected 
from a second non-uniform distribution where the probability 
of selecting one item from the bin was 87%, two items was 
8%, and three items was 5%. 


The human mind typically stores 3-5 items in its short-term 
memory. We employed this method of generating pick lists to 
push the limits of our participants’ short-term memory [4, 7]. 
We posited that the large number of items (4-18 per suborder) 
would make the participants more likely to make a mistake 
than previous experiments. 


Test sessions were video-recorded and timed. Three of the 
methods were timed using an automatic logging system and 
one (Paper-None) was timed with a stopwatch and recorded by 
hand by an attending researcher. After the experiment, partici- 
pants completed a NASA-TLX survey for each pick method 
and an overall ranked-preference survey at the conclusion of 
their participation in the study. 


Labeling and Categorizing Incorrect Orders 

Error checking was performed by taking pictures of the receive 
bins after each task and comparing the receive bins with the 
expected items, as shown in Figure 10. 


We first labelled all the images as correct or incorrect. After 
that, we revisited the incorrect ones and further separated them 
into different categories which are consistent with previous 
literature [13]. 


We define the following categories of errors: 


e Missing Item - An item ordered is not received. 


Insertion - An extra item is added to an order. 


Substitution - Equivalent to a combination of insertion and 
missing item; it is when the wrong item is received in place 
of another item 


Pick too few - The correct source bin is chosen but there 
are not enough items picked. Equal to the number of items 
missing. 


Pick too many - The correct source bin is chosen but there 
are too many items picked. 


ANALYSIS AND RESULTS 

We report average task time, average item error, subjective task 
load, and user preferences for all four pick and verification 
methods. 
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Figure 6. Our experimental environment consists of two racks (named A and B) and a cart. Each rack has 12 source bins, and the cart has three receive 
bins (bottom row not used). 


Subject Learning Curves 
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Figure 9. The time per pick for every task in order. The dashed line 
separates the training from the testing phase. 


Figure 7. Image of experimental environment. , 
Order Error Labeling 
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Figure 8. System architecture. 


Item Errors by Type 
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Figure 12. Breakdown of order errors by type. 
* indicates a statistically significant difference between means. 


Average Task Time 

We used pairwise one-tailed t-tests with Benjamini-Hochberg 
correction for multiple hypothesis testing to test the null hy- 
potheses that the average task time of the HUD-RFID tasks 
was greater than or equal to all other methods. The null hypoth- 
esis was rejected in all cases. In other words, HUD-RFID was 
faster than every other method we tested. Figure 11 compares 
the average task time for each method evaluated. 


Average Task Time 
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Figure 11. Average task time for each method. 


Errors 
In Figure 12, we display the total number of errors for each 
method with a breakdown by error type. 


To test the significance of the error rate between different pick- 
ing methods, we use pairwise one-tailed t-tests with Benjamini- 
Hochberg correction for multiple hypothesis testing to test the 
null hypothesis that the average item error of HUD-RFID is 
greater than the average item error of all other methods. We 
can reject the null hypothesis and conclude that average item 
error of pick-to-HUD with RFID is less than the average item 
error of pick-to-paper with no verification and pick-to-paper 
with barcode verification. 


When comparing HUD-RFID with Light-Button, the differ- 
ence in mean errors is not statistically significant, and the error 
distributions are similar. 


User Preferences 

In the same vein, we made interesting observations on the 
user preferences. We used the one-tailed Wilcoxon Signed- 
Rank test to test the null hypothesis that overall preference 
for HUD-RFID was less than the overall preference for all 
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Figure 13. The absolute differences between our comparison three meth- 
ods and our reference method: Paper-None (average workload = 58). 


other methods. When comparing HUD and Barcode, (W = 0) 
the result was statistically significant. When comparing HUD 
and Paper, (W = 13) the result was statistically significant. 
However, when comparing HUD and Light, (W = 22.5) the 
result was not statistically significant. 


Table 1 outlines average user ratings of our four methods 
through a ranked preference survey. 


HUD-RFID  Light-Button Paper-Barcode Paper-None 
Overall 1.50 2.00 3.92 2.58 
Learnability 1.92 2.00 3.83 2.17 
Comfort 2.42 1.33 3.92 2:33 
Speed 1.33 2.17 3.92 2.50 
Accuracy 1.08 2.17 3.25 3.42 


Table 1. User Preferences 


Overall Task Load 

We employed the NASA Task Load Index to measure the 
workload of each method. The survey quantifies the subjects 
perception of the mental demand, physical demand, temporal 
demand, performance, effort, and frustration with a task. The 
measure ranges from 0 to 100 with larger values indicating 
higher load. Figure 13 details our results. 


We used the one-tailed t-tests with Benjamini-Hochberg cor- 
rection for multiple hypothesis testing to test the null hypoth- 
esis that overall workload for HUD-RFID was higher than 
that for all other methods. All differences in means versus 
HUD-RFID reached statistical significance. 


DISCUSSION AND FUTURE WORK 

In terms of task times, our results are largely consistent with 
the results from previous work. The observed percentage 
improvement in errors compared to Pick-to-Paper was 75% 
for HUD-RFID, 82% for Light-Button, and 64% for Paper- 
Barcode. However, the number of order errors is significantly 
higher than in previous studies as we sought to purposefully 


173 


174 


increase the number of errors in our study to better understand 
what types of errors RFID-based verification can prevent. 


Error Types across Pick Methods 

We proceeded to analyze the individual item errors in terms 
of their type. By breaking down the errors into type, we can 
separate errors into different classes based on severity. For 
example, less severe errors are insertion and pick too many 
because they increase the company cost but do not generate 
customer dissatisfaction. More severe errors include missing 
item, substitution, and pick too few. 


Causes of Errors 

At this point, we wanted to investigate the cause of errors for 
each method. Did the errors result from a lapse of attention, 
an unclear rule, or a systemic issue? How could a subject not 
pick an item in the Light-Button method, for example, when 
clicking the bin button is the only way to proceed to the next 
task? We believed that such an insight would help us find 
faults in our existing solutions. 


None of the methods we used were able detect pick-too-many 
and pick-too-few errors, so we excluded these error types from 
the analysis. 


HUD-RFID 
Most errors recorded for this method were preventable. 


A number of errors were either directly or indirectly linked to 
a faulty RFID band (xBand) which often prevented subjects 
from scanning a certain tag. This problem led subjects to favor 
one hand over the other and had a significant impact on their 
flow and immersion in the process. In one such example, a 
subject reached into a bin and picked a single item. He then 
reached with his other hand into a different bin and scanned 
the tag but stopped before completing the pick as he realized 
that the previous pick was not registered by the faulty RFID 
band. He went back and re-scanned the first bin, but forgot to 
complete the second pick, thus causing a missing item error. 


Another important cause of errors involved subjects placing 
items in the wrong receive bin. In Figure 4 we presented 
our attempt to aid the subjects for correcting such errors on 
the spot, before they impact the end customer. Our method 
helped prevent a number of such errors, but was not optimal. 
The process required subjects to remove all misplaced items 
from the wrong receive bin before placing them in the correct 
one. There were issues, however, when subjects attempted to 
correct such errors in batches which would cause our system 
to proceed after the first place and would not allow them to 
correct the rest. 


Finally, a small number of errors were caused when the RFID 
bands incorrectly scanned a tag under the desired bin. This 
problem was caused by the variable scan distance of the bands, 
possibly influenced by metal objects in their environment. In 
one such example, a subject had to pick items from bin B43, 
but incorrectly reached into bin B33 which is directly on top. 
The RFID band registered and scanned both tags which led 
the subject to believe he made the correct pick. 


Light-Button 
In Light-Button, errors were more straightforward. 


Many errors were caused by lapses in attention. Subjects 
would click one button and pick from a different bin. This type 
of error was surprisingly common because the vast majority of 
subjects pushed buttons with one hand and picked items with 
the other, which increased the probability of a human error. 
Such errors would not affect HUD-RFID due to the fact that 
the hand that scans is also the hand that picks. 


The rest of the errors were wrong receive bin errors. Light- 
Button has no way of correcting these errors, so they took a 
toll even if the subject recognized the mistake immediately. 


Paper-Barcode 

In Paper-Barcode, many errors were caused by the cognitive 
distance between verification and pick. Subjects would scan a 
tag and not complete a pick, or would scan a number of tags 
in a batch and would only pick a portion of them. 


A number of errors were also caused by an inherent problem 
of our implementation. Our barcode scanner did not include a 
display and thus could not give subjects a clear status of their 
progress. Subjects would miss a pick, and the system would 
not allow them to proceed. When that happened, subjects 
could not easily find which items they were missing. Some 
proceeded to scan every tag in sequence, which generated a 
number of missing item errors. In future study design, we 
could improve the barcode system by adding a display to the 
scanner device. 


Paper-None 

Because pick-to-paper offers no verification, most errors were 
straightforward. Subjects would skip some picks, perform 
others incorrectly and some would even pick from the wrong 
racks all together. 


Substitution errors are understandably higher than in the other 
three methods that offer some form of confirmation for users. 
Any form of verification adds a second, system-enforced check 
on the user’s behavior and will reduce substitution errors. 


One of our main takeaways is that the further one distances 
the verification from the pick, either in time or space, the more 
human errors will be induced. This result is promising for the 
HUD-RFID method which has the verification embedded in 
the pick. 


Subject Tactics to Overcome Technological Limitations 


HUD-RFID 

We observed users taking time to find the optimal way to move 
their hands in order to find the tags. After training, users 
would be able to scan the correct bins and would have less 
unintended scans with other bins near the vicinity. 


Light-Button 

We also observed that pickers typically step back to view the 
entire shelving unit and then scan the entire rack from top to 
bottom to find bins with remaining items. This observation is 
consistent to behaviors observed in Guo, et al. [5]. 


Paper-Barcode 

We observed that subjects struggled to hold both the paper-list 
and the scanner whilst picking items. Some subjects attempted 
to free one hand for picking by holding both the barcode 
scanner and paper in one hand. Others collected items on 
their clipboard and then deposited them into the receive bin 
by sliding them off. Finally, some would first scan several 
bins, memorize them and then pick all the items at once. The 
fastest subject decided to hang the clipboard from the rack and 
picked the items with the free hand. 


Post-hoc Observations 

A surprising post-hoc observation was the difference between 
Pick-to-Paper with Barcode and Pick-to-Paper with No ver- 
ification. We anticipated that Pick-to-Paper with Barcode 
would be extremely similar to Pick-to-Paper in terms of time - 
barcode scan actions are generally thought of as quick. How- 
ever, there is a statistically significant difference in the timings 
(p < 0.001). 


It would be interesting to investigate further into wearable 
barcode scanners to see if it could minimize the time taken for 
Pick-to-Paper with Barcode. 


CONCLUSION 

We explored a novel wearable RFID-based verification method 
to understand the per task speed and accuracy improvements 
of HUD-RFID compared to standard methods. We found 
that HUD-RFID was faster than all other methods and that 
errors occurred significantly less when compared to pick-by- 
paper with no verification and pick-by-paper with barcode 
verification. In terms of overall preference and workload, 
HUD-RFID was the most preferred and offered clear usability 
and comfort benefits. Considering the high implementation 
costs of pick-to-light, the high error rate of pick-to-paper, 
and the discomfort of pick-to-paper with barcode verification, 
pick-to-HUD with RFID verification offers a strong and cost- 
effective solution to fast and accurate order picking. 
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